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contribution  of  the  Study  Advisory  Group  (SAG).  This  group  of  Quality  Assurance  experts 
from  throughout  the  Defense  Contract  Management  Command  provided  valuable  assistance  in 
designing  the  model  and  generating  subjective  weighting  factors  for  consistency  in  the  metrics. 
We  are  very  grateful  for  their  enthusiastic  support  throughout  this  effort. 

This  report  is  intended  to  document  the  development  of  QUEST  4.0.  It  was  written  to  also  serve 
as  a  guide  to  users,  new  and  former,  in  understanding  how  QUEST  ratings  are  computed  and 
presented.  More  detailed  documentation  is  available  upon  request. 
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SECTION  1 
INTRODUCTION 


1.1  BACKGROUND 

1.1.1  QUEST  1.0 

Quality  Effectiveness  Sensing  Technique  (QUEST)  was  developed  by  the  Defense  Logistics 
Agency  (DLA)  Operations  Research  Office  (DORO)  starting  in  1986,  with  deployment 
throughout  the  nine  Defense  Contract  Administrative  Services  districts  in  1988-1989.  QUEST 
Release  1.0  provided  two  sets  of  basic  measures;  one  set  measured  in-house  program  compliance 
with  the  then  existing  Contract  Quality  Assurance  Program  (CQAP)  and  the  other  set  measured 
the  contractors  conformance  to  contract  requirements.  For  more  information,  documentation 
was  published  under  DLA-LO  Project  3071,  Analysis  of  Quality  Assurance  (QA)  Effectiveness. 

1.1.2  QUEST  2.0 

Release  2.0,  fielded  in  1990,  involved  minor  changes  to  QUEST  1.0  but  significantly  reduced 
the  cost  of  distributing  the  reports  by  incorporating  electronic  distribution  vice  hard  copy 
(reference  project  DLA-90-P90271,  Quality  Effectiveness  Sensing  Technique  (QUEST)  Release 
II). 


1.1.3  QUEST  3.0 

When  CQAP  was  replaced  by  the  In-plant  Quality  Evaluation  Program  (IQUE),  QUEST  2.0 
became  obsolete  and  was  modified  to  reflect  the  new  policy.  The  in-house  measurement  system 
was  removed  and  QUEST  3.0  became  strictly  a  tool  for  evaluating  total  contractor  effectiveness. 
With  the  teamwork  concept  associated  with  IQUE,  it  was  concluded  that  contractor  effectiveness 
is  a  surrogate  measure  of  IQUE  total  program  effectiveness.  QUEST  3.0  was  completed  in 
March  1991  under  DLA-91-P90272,  Quality  Effectiveness  Sensing  Technique  (QUEST) 
Release  3.0. 

1.1.4  QUEST  4.0  (CORPORATE  QUEST) 

Since  QUEST  is  totally  dependent  on  the  Quality  Assurance  Management  Information  System 
(QAMIS)  for  data,  QUEST  must  be  modified  each  time  QAMIS  changes  structure.  With  a 
recent  major  change  to  QAMIS,  a  new  version  of  QUEST  was  needed.  QUEST  4.0  will  take 
advantage  of  some  newly  added  data  elements  to  QAMIS  to  provide  better  measures  of 
contractor  quality  assurance  effectiveness. 

1.2  SCOPE 

QUEST  4.0  will  provide  measures  of  contractor  quality  assurance  effectiveness  for  all 
contractors  reporting  data  in  QAMIS.  Thus  nearly  all  contractors  under  Defense  Contract 
Management  Command  (DCMC)  will  be  measured.  Excluded  contractors  are  those 


administered  by  DCMC-Intemational  (foreign  contractors),  inactive  contractors,  and  contractors 
with  insufficient  DCMC  activity  to  reasonably  produce  a  measure.  For  the  fn.  t  time,  QUEST 
4.0  will  also  provide  "corporate"  measures  of  effectiveness  for  large  defense  contractors  by 
combining  individual  facility  measures  nationwide  since  most  large  contractors  have  production 
facilities  in  several  DCMC  districts. 

1.3  OBJECTIVES 

To  develop  measures  of  contractor  effectiveness  for  quality  assurance  at  various  levels  in  the 
DCMC  quality  organization  structure  and  contractor  reorganization  structure.  Measures  must 
begin  at  the  contractor  facility  level  and  roll  up  according  to  volume  of  business. 

To  provide  access  to  QUEST  measures  from  the  first  line  supervisor  level  to  upper  management 
in  DLA. 


I 
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SECTION  2 
METHODOLOGY 


2.1 

DA IA 

2.1.1 

DATA  SOURCES 

QUEST  4.0,  like  its  predecessors  draws  upon  data  files  available  in  the  Mechanization  of 
Contract  Administrative  Service  (MOCAS)  system.  The  accuracy  of  QUEST  measures  is 
heavily  dependent  on  the  accuracy  of  the  data  in  these  files.  There  is  one  file  which  is  the 
primary  source  of  QUEST  data  named  the  Performance  History  File,  a  subset  of  QAMIS.  This 
file  is  comprised  of  monthly  data  (12  months)  reported  by  the  Quality  Assurance  Representative 
(QAR)  through  the  QAMIS.  Most  of  the  data  fields  used  by  QUEST  in  the  Performance  History 
File  are  believed  to  be  relatively  accurate  by  the  Study  Advisory  Group  (SAG).  In  addition  to 
the  Performance  History  File,  other  data  sources  used  by  QUEST  are: 

Facility  Profile  -  Another  subset  to  QAMIS  provides  descriptive  information  on  the 
contractor. 

Product  Quality  Deficiency  Report  (PQDR)  File  (YP21)  -  MOCAS  file,  source  of  data 
for  PQDR  portion  of  QUEST  rating. 

Contractor  Alert  File  -  MOCAS  file  used  to  help  identify  contractors  that  are 
experiencing  difficulties  in  contract  administration. 

ADRS  File  -  MOCAS  file  used  to  associate  "in  the  clear”  the  name  of  the  contractor  and 
the  cognizant  contract  administration  organization  for  report  distribution.  The  ADRS  file 
contains  administrative  information  on  contractors  by  Commercial  and  Government 
Entity  Code. 


2.1.2  EXCLUSIONS 

Contractors  are  excluded  from  QUEST  if  their  number  of  active  months  in  the  last  12  months  is 
smaller  than  the  model  parameter.  The  default  model  parameter  is  currently  3  active  months. 
QUEST  scores  are  only  generated  for  the  contractor's  active  months.  A  month  is  considered 
active  if  during  that  month,  a  PDQR  was  closed  or  in-house  hours  were  reported  against  the 
contractor  as  identified  by  a  Commercial  and  Government  Entity  Code  (CAGE).  For  example, 
if  there  were  less  than  three  months  of  activity  in  a  CAGE  within  the  last  twelve  months,  the 
CAGE  is  considered  inactive  and  will  not  be  measured  at  all.  If  there  were  three  or  more 
months  of  activity  in  a  semi-active  nonresident  CAGE,  QUEST  scores  will  be  generated  only 
during  a  month  where  there  was  activity.  For  active  nonresident  and  resident  contractors, 
QUEST  should  produce  a  set  of  measurements  each  month. 
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2.1.3 


DATA  LINKING 


AH  data  sources  used  by  QUEST  arc  linked  together  by  CAGE  Code.  In  QAMIS  some  large 
contractors  which  have  a  single  CAGE  Code  arc  broken  down  into  more  discrete  entities  using 
"locally  assigned"  CAGE  codes.  This  may  cause  some  difficulties  in  the  measurement  process  if 
all  the  source  files  do  not  employ  the  same  "locally  assigned"  CAGE  codes  as  the  Performance 
History  File.  For  example,  if  all  PQDRs  are  reported  against  the  original  CAGE,  the  PQDR 
QUEST  rating  for  the  "locally  assigned"  CAGES  will  be  inflated  and  the  QUEST  rating  for  the 
original  CAGE  will  be  too  low. 

2.2  QUEST  INDICATORS 

Traditionally  with  DLA,  quality  of  product  is  not  measured  directly  but  rather  through  the  use  of 
"negative"  indicators  such  as  complaints,  deficiency  reports  and  other  signals  that  there  are 
nonconformance  problems.  Likewise,  QUEST  4.0  is  an  amalgam  of  negative  quality  indicators. 
Contractors  that  have  a  high  incidence  rate  of  negative  quality  indicators  will  fare  poorly  with 
QUEST.  Absence  of  negative  quality  indicators  will  yield  beneficial  QUEST  ratings.  The 
seven  components  are  described  as  follows,  in  order  of  importance  to  the  QUEST  score: 

2.2. 1  CORRECTIVE  ACTION  REQUESTS  (CARs) 

When  a  contract  nonconformance  is  observed  the  QAR  is  supposed  to  generate  a  CAR.  There 
are  four  levels  of  CAR  (verbal  and  written.  Method  C,  Method  D,  and  Method  E).  Each  level  is 
weighted  in  the  model  and  then  summed  to  produce  a  CAR  indicator  value.  Weights  for  CARs 
and  other  indicators  were  determined  by  the  SAG  and  can  be  found  in  a  separate  DORO 
publication  entitled  QUEST  4.0  Documentation. 

2.2.2  PRODUCT  QUALITY  DEFICIENCY  REPORTS  (CUSTOMER 

COMPLAINTS) 

When  an  investigated  PQDR  is  closed  (a  new  feature  of  Release  4.0,  previous  releases  measure 
PQDR  based  on  date  received),  and  if  the  PQDR  is  determined  to  be  caused  bv  the  contractor. 
the  PQDR  is  considered  valid.  Valid  PQDRs  are  weighted  by  the  age  of  the  contract  since  more 
recent  contracts  are  prone  to  reflect  current  processes  within  the  plant. 

2.2.3  PRODUCT  AUDIT  RESULTS 

The  ratio  of  Product  Audits  with  CARs  to  total  Product  Audit  is  computed  to  represent  the 
proportion  of  time  nonconforming  product  was  released  for  government  inspection. 

2.2.4  WAIVERS 

Waivers  are  requested  changes  to  contract  requirements  generated  during  production.  The 
weighted  sum  of  Type  I  and  Type  II  Waivers  is  computed.  This  is  a  new  feature  since  the  prior 
QAMIS  only  reported  total  waivers.  Per  DLAM  8200.5,  a  Type  I  nonconformance  affects 
"performance,  durability,  interchangeability,  effective  use  or  operations,  weight  or  appearance 
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(where  a  factor),  health  or  safety."  Type  II  are  all  other  nonconformances.  Because  Type  I 
waivers  are  more  consequential,  they  carry  more  weight  in  assessing  performance. 

2.2.5  MATERIAL  REVIEW  BOARD  (MRB)  ACTIONS 

Nonconformances  that  do  not  affect  form,  fit,  or  function  are  often  reported  through  an  MRB 
procedure.  The  total  MRB  actions  reported  in  a  month  is  used  to  assess  performance. 

2.2.6  DEVIATIONS 

Deviations  are  requested  changes  to  contract  requirements  generated  prior  to  production.  The 
number  of  deviations  reported  in  a  month  is  used  to  assess  performance. 

2.2.7  ENGINEERING  CHANGE  PROPOSALS  (ECPs) 

ECP  is  a  process  used  to  change  the  specifications  on  the  current  contract  and  future  contracts. 
The  weighted  sum  of  Class  I  and  Class  II  ECP  is  computed.  This  is  a  new  feature  of  Release  4.0 
since  early  QAMIS  data  only  reported  total  ECP  counts.  The  definition  of  Class  I  is  provided  in 
DLAM  8200.5,  listing  five  major  criteria  and  twenty-one  subcriteria.  Generally,  Class  I  ECPs 
are  more  technically  significant  and  therefore  are  weighted  more  than  Class  II  by  QUEST  4.0. 

2.3  INDICATOR  MEASUREMENT 

Each  of  the  seven  indicators  described  in  the  previous  section  is  measured  on  a  common  scale  of 
0  to  100  percent  using  a  method  called  Technique  for  Order  Preference  by  Similarity  to  Ideal 
Solution  (TOPSIS).  TOPSIS  is  used  to  reduce  a  multi-dimensional  variable  to  a  single  value.  In 
QUEST  each  indicator  has  two  variables  of  interest,  namely  trend  and  peer  comparison. 
Depending  on  the  combination  of  trend  and  peer  comparison,  a  score  is  computed  where  100 
percent  represents  the  best  possible  combination  of  variables  (ideal  condition)  and  0  percent 
represents  the  worst  case  (negative  ideal).  TOPSIS  measures  each  contractor  against  these  fixed 
reference  points,  creating  a  percent  rating  which  is  the  distance  of  the  contractor's  trend  and  peer 
comparison  from  the  negative  ideal  to  the  total  distance  from  both  ideal  and  negative  ideal. 
Therefore,  for  example,  a  contractor  with  an  80  percent  rating  for  a  given  indictor  means  that 
the  contractor’s  distance  from  the  worst  case  is  80  percent  of  the  total  distance  from  both 
reference  points.  The  higher  the  distance  from  the  worst,  the  better  the  contractor's  rating. 

Thus,  QUEST  measurement,  from  the  perspective  of  the  contractor,  creates  an  incentive  for 
continuous  improvement  and  "best  of  class"  performance. 

2.3.1  TREND  MEASUREMENT 

QUEST  computes  a  trend  by  looking  back  up  to  six  months  (less  than  six  month  trend 
computation  is  used  for  semi-active  nonresidents  when  six  months  of  historical  data  is  not 
available).  The  six  month  interval  is  divided  into  two  equal  intervals  (recent  and  less  recent)  and 
the  average  indicator  value  is  taken  for  each  interval.  The  trend  is  computed  from  the  two 
averages  and  normalized  on  a  scale  of  -3.0  to  +3.0.  Minus  three  is  the  ideal  condition, 
representing  a  steeply  decreasing  trend  (since  indicators  are  negative  indicators,  less  is  best). 
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Plus  three,  a  steep  rise  in  trend,  is  the  worst  case  or  negative  ideal.  Zero  trend  is  a  flat  trend, 
indicating  neither  improvement  nor  worsening  of  the  indicator. 

Trend  contributes  half  the  total  weight  in  computing  a  QUEST  rating  for  each  indicator.  Peer 
comparison  contributes  the  other  half.  There  are  two  exceptions  to  this  rule.  The  first  is  for 
contractors  that  have  a  totally  clean  record  on  an  indicator.  They  have  no  recorded  incidence 
within  the  measurement  period  (currently  1  year).  The  QUEST  indicator  score  is  automatically 
set  to  100  percent  for  any  contractor  that  has  no  history  for  the  indicator.  Otherwise  QUEST 
would  compute  a  flat  trend  and  penalize  the  contractor  relative  to  other  contractors  that  were 
experiencing  the  indicator  yet  improving.  The  second  exception  for  the  50/50  rule  is  for 
"problem"  contractors.  Trend  for  problem  contractors  receives  60  percent  of  the  weight  for  all 
indicators. 

"Problem"  contractors  are  defined  by  QUEST  as  any  contractor  that  meets  at  least  two  of  the 
following  conditions. 

-Is  currently  on  the  Contractor  Alert  list  for  any  reason. 

-Has  more  than  one  standard  deviation  above  average  in  PQDRs  (complaints)  above 
peers. 

-Has  received  a  method  C  CAR  within  the  last  three  months. 

-Has  received  a  method  D  CAR  within  the  last  year. 

-Has  received  a  method  E  CAR  within  the  last  three  months. 

2.3.2  PEER  COMPARISON 

The  second  component  of  an  indicator  measurement  relates  to  how  the  contractor’s  reported 
indicator  compares  with  preset,  fixed  statistics  on  similar  contractors  taken  for  calendar  year 
1992.  QUEST  converts  the  monthly  indicator  to  a  scale  of  -3.0  to  +3.0  where  zero  is  the  peer 
group  average;  the  converted  value  represents  how  many  standard  deviations  the  contractor  falls 
below  (minus  values)  or  above  (plus  deviates)  the  peer  group  average.  Minus  three  is  the  ideal 
condition  and  plus  three  is  the  negative  ideal  for  TOPSIS  measurement. 

Peer  groups  are  determined  primary  by  Facility  Profile  data  by  assigning  a  peer  group  number 
between  1  and  999.  Each  peer  group  generally  has  the  following  factors  in  common. 

Facility  Type  -  Resident  or  Nonresident 

Commodity  -  Per  DLAM  8200.2 

Provision  -  MIL  Q,  MIL  I  or  Other 

Operation  Type  -  Manufacturer  or  Maintenance 
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Size  (Resident  Facilities)  -  Based  on  the  number  of  Government  QA  Specialists 

(QAS)  which  is  determined  by  the  available  hours 
reported  in  the  Performance  History  File.  Size  groups  are 
0-2,  3-7,  8-20,  21-35,  and  over  35. 

In  some  instances,  to  obtain  sufficient  numbers  of  similar  contractors,  commodity  codes  are 
combined.  A  separate  report  is  available  which  lists  contractors  nationwide  by  peer  group. 


2.4  QUEST  RATINGS 

After  each  indicator  is  measured,  an  overall  rating  is  formed  by  taking  a  weighted  average  of  the 
individual  indicators.  The  weights  were  determined  by  the  SAG  to  be  as  shown  in  Table  2-1. 

Table  2-1  .  Weight  Factors  for  QUEST  Indicators 


FACTOR 

WEIGHT  % 

CAR 

22 

PQDR 

21 

Product  Audit 

20 

Waivers 

15 

MRB 

11 

Deviations 

7 

ECP 

_4 

100 

In  addition  to  a  numeric  rating,  a  letter  grade  of  A,  B,  C,  D,  or  F  is  assigned  by  comparing  the 
numeric  rating  to  peer  group  statistics.  QUEST  ratings  of  100  percent  or  those  that  are  well 
above  peer  averages  translate  into  an  "A"  grade;  in  like  manner,  very  low  numeric  ratings 
compared  to  peer  averages  result  in  an  "F"  rating.  A  "C"  rating  is  about  average. 

2.5  ORGANIZATIONAL  ROLL  UPS 

QUEST  numeric  scores  are  rolled  up  to  compute  averages  for  each  level  based  on  the  QA 
Organization  Code.  Tims,  section,  branch,  division  and  district  level  scores  are  available  each 
month.  Organizational  averages  are  computed  by  factoring  each  CAGE  (contractor)  according 
to  the  size  of  the  contractor  as  measured  by  the  hours  of  reported  time  attributed  by  the 
government  QASs'  assigned.  Therefore,  large  contractors  contribute  more  to  organizational  roll 
ups  than  small  contractors.  Section  roll  ups  also  breakout  the  rating  by  resident  versus 
nonresident  contractors  within  the  section. 

2.6  CORPORATE  QUEST 

Ratings  are  now  available  for  large  Defense  contractors  that  cross  district  boundaries.  A 
separate  report  is  available  which  consolidates  major  Defense  contractor's  QUEST  ratings  by 
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CAGE  into  an  overall  letter  grade.  The  Corporate  grade  is  equivalent  to  a  Grade  Point  Average 
(GPA)  generated  by  a  university.  Individual  CAGE  letter  grades  are  converted  to  a  4.0  point 
scale  (A  *  4  points,  B  =  3  points,  etc)  and  a  corporate  cumulative  average  is  computed  by 
factoring  in  the  size  of  the  CAGE.  Just  like  some  college  courses  carry  more  credit  hours  and 
thus  contribute  more  to  the  GPA;  a  larger  facility  in  a  corporation  will  contribute  more  to  the 
corporate  average  than  a  smaller  one. 

2.7  QUEST  VALIDATION 

As  previous  versions  of  QUEST  were  tested  prior  to  implementation,  the  SAG  advised  that 
because  of  the  significant  changes  associated  with  Release  4.0,  revalidation  was  required.  A 
Mann-Whitney  hypothesis  test  was  designed  whereby  QUEST  scores  were  compared  against 
expert  opinion  concerning  the  Quality  Assurance  performance  of  a  set  of  contractors. 
Fifty-eight  resident  facilities  and  60  nonresident  facilities  were  judged  by  experts  to  be  either 
"excellent"  or  "poor"  concerning  quality.  The  degree  with  which  expert  opinion  matched  to 
QUEST  ratings  was  measured  and  evaluated  at  the  95  percent  confidence  level. 


2-6 


SECTION  3 
FINDINGS 

3.1  QUEST  REPORTS 

QUEST  reports  are  available  monthly  through  hard  copy  printouts  generated  by  the 
Mechanization  of  Reports  Distribution  System  to  the  division  level.  On-line  access  is  also 
available  through  the  DMINS  system.  The  hard  copy  report,  shown  as  Table  3-1,  is  similar  to 
previous  reports.  The  only  two  changes  are  that  a  six  month  moving  average  is  shown  for  the 
QUEST  score  and  facilities  that  have  experienced  a  change  in  peer  group  are  identified  with  a 
"#"  after  the  peer  group  number.  An  explanation  of  the  column  headings  of  Table  3-1  is 
provided  for  users  unfamiliar  with  the  earlier  versions  of  the  report.  A  similar  report  for 
Corporate  ratings  is  shown  in  Table  3-2.  Corporate  reports  are  currently  available  for  HQ  DLA 
users  of  the  DLA  Operations  Research  Analysis  Network. 

3.1.1  PAGE  HEADERS 

Unless  a  section  has  more  than  50  separate  contractors,  each  section  will  receive  a  one  page 
summary  of  QUEST  information.  The  page  header  contains  the  year  and  month,  the  QA 
Organization  Code  and  the  Contract  Administration  Code. 

3.1.2  COLUMN  HEADINGS 
The  following  columns  appear: 

CAGE:  Commercial  and  Government  Entity  of  the  contractor 

NAME:  In  the  clear  name  of  the  contractor 

QAR:  Code  of  the  QAR  assigned  to  the  contractor 

COMM:  Primary  Commodity  Code 

PVN:  Highest  QA  Provision  for  active  contracts 

QAS:  Number  of  QA  Specialist-function  of  hours  reported 

CA:  CAR  Score 

PQDR:  PQDR  Score 

PA:  Product  Audit  Score 

WVRS:  Waiver  Score 

MRB:  Material  Review  Board  Score 

DEVN:  Deviation  Score 

ECP:  Engineering  Change  Proposal  Score 

TOTAL  SCORE:  QUEST  Score  for  current  month 

FIRST  PRIOR  MONTH:  QUEST  Score  for  last  active  month 

SECOND  PRIOR  MONTH:  QUEST  Score  for  next  to  last  active  month 

SIX  MONTH  AVG:  Average  of  most  recent  six  active  months 

PEER  GRP:  Peer  Group  number  assigned  by  QUEST 

PEER  RTG:  QUEST  rating  relative  to  peers 
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If  a  section  has  both  resident  and  nonresident  contractors,  the  resident  appear  first  followed  by  a 
blank  line.  The  order  of  appearance  of  contractors  is  by  increasing  QUEST  Scores  for  the 
current  month.  Problem  contractors  are  designated  with  an  before  the  CAGE  Code.  If  a 
contractor  has  changed  peer  groups,  a  appears  after  the  Peer  Group  number. 

Roll  ups  by  organization  appear  at  the  bottom  of  each  page.  Section  roll  ups  are  weighted 
averages  of  all  contractors  within  a  section;  Branch  roll  ups,  the  weighted  average  of  contractors 
within  the  branch,  appear  after  the  last  section  roll  up,  etc.  Roll  ups  are  available  through  the 
district  level. 
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Table  3-2.  Sample  Corporate  QUEST  Report 


DISTRICT 

CAGE 

CORPORATE  NAME 

CA 

POOR 

PA 

WVRS 

MRB 

DEVN 

ECP 

OUEST 

RATING 

NQAR 

M 

CHBH2 

HUGHES  DISPLAY  PRO 

100.0 

19.6 

20.0 

100.0 

100.0 

50.0 

100.0 

51.0 

D 

0 

M 

21538 

HUGHES  INDUSTRIES 

100.0 

21.0 

60.0 

100.0 

100.0 

0.0 

100.0 

100.0 

D 

0 

V 

2F259 

HUGHES  AIRCRAFT  CO 

100.0 

77.8 

40.0 

100.0 

100.0 

99.8 

100.0 

100.0 

A 

1 

W 

3U331 

HUGHES  AIRCRAFT  CO 

100.0 

40.4 

40.0 

22.6 

100.0 

73.5 

100.0 

100.0 

C 

4 

v 

15090 

HUGHES  AIRCRAFT  CO 

50.0 

78.7 

60.0 

48.7 

100.0 

100.0 

100.0 

100.0 

A 

23 

V 

1181M 

HUGHES  AIRCRAFT  CO 

100.0 

100.0 

70.0 

53.0 

100.0 

100.0 

100.0 

100.0 

A 

2 

w 

05889 

HUGHES  AIRCRAFT  CO 

45.9 

53.1 

55.0 

42.6 

100-0 

70.8 

71.9 

100.0 

C 

7 

•  w 

00816 

HUGHES  AIRCRAFT  CO 

100.0 

100.0 

80.0 

100.0 

100.0 

100.0 

100.0 

100.0 

A 

O 

w 

00GR4 

HUGHES  ENTERPRISE 

100.0 

100.0 

100.0 

100.0 

100-0 

100.0 

100.0 

100.0 

A 

0 

w 

53669 

HUGHES  AIRCRAFT  CO 

100.0 

100.0 

60.0 

100.0 

100.0 

100.0 

100.0 

100.0 

A 

o 

w 

53670 

HUGHES  AIRCRAFT  CO 

100.0 

100.0 

80.0 

100.0 

100.0 

100.0 

100.0 

100.0 

A 

0 

v 

55267 

HUGHES  AIRCRAFT  CO 

100.0 

54.7 

40.0 

100.0 

100.0 

54.8 

100.0 

100.0 

C 

2 

w 

7238E 

HUGHES  AIRCRAFT  CO 

70.6 

58.8 

55.0 

54.4 

50.0 

61.4 

16.6 

50.0 

F 

6 

w 

7239J 

HUGHES  AIRCRAFT  CO 

87.2 

74.6 

70.0 

39-.  3 

25.5 

54.3 

42.1 

50.0 

F 

3 

CORPORATE  TOTAL 

B 

DISTRICT 

CAGE 

CORPORATE  NAME 

CA 

POOR 

PA 

WVRS 

MRB 

DEVN 

ECP 

OUEST 

RATING 

NOAR 

C 

76301 

MCDONNELL  DOUGLAS 

44.1 

39.9 

45.0 

18.7 

25.8 

69.1 

40.0 

100.0 

F 

46 

C 

69236 

MCDONNELL  DOUGLAS 

24.7 

55.3 

55.0 

29.0 

18.7 

73.9 

100.0 

64.7 

F 

6 

C 

0ASG3 

MCDONNELL  DOUGLAS 

100.0 

74.2 

55.0 

31.3 

100.0 

69.0 

100-0 

100.0 

C 

4 

C 

0FC43 

MCOONNELL  DOUGLAS 

53.0 

60.0 

55.0 

47.5 

100.0 

38.6 

10.2 

100.0 

D 

5 

C 

12464 

MCDONNELL  DOUGLAS 

79.8 

45.8 

55.0 

22.9 

100.0 

65.4 

100.0 

100.0 

C 

5 

M 

0D7A6 

MCDONNELL  DOUGLAS 

100.0 

100.0 

55.0 

31.4 

100.0 

100.0 

100.0 

100.0 

A 

2 

S 

0KPR9 

MCOONNELL  DOUGLAS 

72.7 

100.0 

76.9 

100.0 

32.3 

71  .4 

100.0 

79.7 

D 

0 

s 

0002 A 

MCDONNELL  DOUGLAS 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

A 

0 

s 

0O32A 

MCDONNELL  DOUGLAS 

63.3 

100.0 

100.0 

100.0 

65.4 

100.0 

38.0 

85.5 

C 

5 

s 

88314 

MCOONNELL  DOUGLAS 

100.0 

19.6 

100.0 

100.0 

77.2 

100.0 

87.2 

79.8 

F 

4 

s 

88277 

MCDONNELL  DOUGLAS 

100.0 

33.7 

45.0 

13.0 

100.0 

34.3 

1000 

100.0 

F 

29 

s 

28861 

MCDONNELL  DOUGLAS 

40.2 

100.0 

44.8 

84.0 

77.4 

100.0 

92.2 

70.6 

F 

6 

•  V 

18355 

MCOONNELL  DOUGLAS 

50.0 

49.6 

55.0 

23.1 

100.0 

34.4 

100.0 

100.0 

C 

9 

V 

1199M 

MCOONNELL  DOUGLAS 

100.0 

60.6 

40.0 

100.0 

100.0 

100.0 

100.0 

100.0 

A 

8 

w 

8V613 

MCDONNELL  DOUGLAS 

54.  1 

88.5 

60.0 

63.5 

100.0 

19.6 

5.  1 

13.2 

F 

23 

CORPORATE  TOTAL  0 
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3.2  VALIDATION  FINDINGS 

Significant  correlation  was  achieved  at  the  DCMC  level  for  both  resident  and  nonresident 
facilities  when  comparing  QUEST  scores  to  expert  opinion.  The  Mann- Whitney  test  statistic  for 
nonresidents  was  -5.07  and  -2.533  for  residents,  resulting  in  Type  I  error  of  nil  and  1.2  percent 
respectively.  At  the  district  level,  most  comparisons  also  resulted  in  significant  correlation  but 
not  always. 

QUEST  4.0  is  the  only  truly  objective,  automated  and  comprehensive  measure  of  contractor  QA 
performance  available.  As  such,  it  should  be  useful  to  DCMC  managers  to  evaluate  the 
effectiveness  of  contractors,  their  IQUE  programs  and  their  policies.  Also,  QUEST  could  be 
used  as  a  possible  tool  for  resource  allocation.  Management's  attention  should  be  directed  to 
contractors  and  sections  which  continually  exhibit  low  QUEST  scores. 
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SECTION  4 
CONCLUSIONS 


QUEST  4.0  is  a  tool  for  evaluating  Contractor  QA  performance.  It  provides  management 
visibility  on  the  overall  effectiveness  and  subcomponents  from  the  plant  level  to  the  corporate 
level  and  from  the  QAR  level  to  the  district  level.  The  measurement  is  based  on  objective  data 
and  subjective  weights  provided  by  experts.  The  results  are  reliable  and  verified  by  statistical 
analysis. 
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SECTION  5 

RECOMMENDATIONS 


We  recommend  that  QUEST  4.0  be  implemented  by  DCMC.  A  major  part  of  implementation 
should  be  a  concerted  effort  by  the  Study  Advisory  Group  to  develop  and  execute  a  QUEST 
training  program  for  managers  and  other  potential  users.  This  training  program  should  stress 
potential  beneficial  applications  by  DCMC  personnel  to  increase  the  use  and  acceptance  of 
QUEST. 

We  also  recommend  that  Contract  Management  Directorate  at  HQ  DLA  appoint 
points-of-contact  to  be  trained  to  extract  QUEST  and  Corporate  QUEST  data  for  senior 
managers  until  such  time  as  QUEST  data  becomes  incorporated  into  the  Executive  Information 
System. 

Finally,  we  recommend  that  analytical  resources  be  planned  for  the  inevitable  update,  QUEST 
5.0.  History  indicates  a  need  to  revise  this  model  about  every  two  years.  FY  95  resources 
should  be  allocated  to  this  effort. 
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